Predictive Maturity of Multi-Scale Simulation Models for Fuel Performance

The project proposed to provide a Predictive Maturity Framework with its companion metrics that (1) introduce a formalized, quantitative means to communicate information between interested parties, (2) provide scientifically dependable means to claim completion of Validation and Uncertainty Quantification (VU) activities, and (3) guide the decision makers in the allocation of Nuclear Energy’s resources for code development and physical experiments. The project team proposed to develop this framework based on two complimentary criteria: (1) the extent of experimental evidence available for the calibration of simulation models and (2) the sophistication of the physics incorporated in simulation models. The proposed framework is capable of quantifying the interaction between the required number of physical experiments and degree of physics sophistication. The project team has developed this framework and implemented it with a multi-scale model for simulating creep of a core reactor cladding. The multi-scale model is composed of the viscoplastic self-consistent (VPSC) code at the meso-scale, which represents the visco-plastic behavior and changing properties of a highly anisotropic material and a Finite Element (FE) code at the macro-scale to represent the elastic behavior and apply the loading. The framework developed takes advantage of the transparency provided by partitioned analysis, where independent constituent codes are coupled in an iterative manner. This transparency allows model developers to better understand and remedy the source of biases and uncertainties, whether they stem from the constituents or the coupling interface by exploiting separate-effect experiments conducted within the constituent domain and integral-effect experiments conducted within the full-system domain. The project team has implemented this procedure with the multiscale VPSC-FE model and demonstrated its ability to improve the predictive capability of the model. Within this framework, the project team has focused on optimizing resource allocation for improving numerical models through further code development and experimentation. Related to further code development, we have developed a code prioritization index (CPI) for coupled numerical models. CPI is implemented to effectively improve the predictive capability of the coupled model by increasing the sophistication of constituent codes. In relation to designing new experiments, we investigated the information gained by the addition of each new experiment used for calibration and bias correction of a simulation model. Additionally, the variability of ‘information gain’ through the design domain has been investigated in order to identify the experiment settings where maximum information gain occurs and thus guide the experimenters in the selection of the experiment settings. This idea was extended to evaluate the information gain from each experiment can be improved by intelligently selecting the experiments, leading to the development of the Batch Sequential Design (BSD) technique. Additionally, we evaluated the importance of sufficiently exploring the domain of applicability in experiment-based validation of high-consequence modeling and simulation by developing a new metric to quantify coverage. This metric has also been incorporated into the design of new experiments. Finally, we have proposed a data-aware calibration approach for the calibration of numerical models. This new method considers the complexity of a numerical model (the number of parameters to be calibrated, parameter uncertainty, and form of the model) and seeks to identify the number of experiments necessary to calibrate the model based on the level of sophistication of the physics. The final component in the project team’s work to improve model calibration and validation methods is the incorporation of robustness to non-probabilistic uncertainty in the input parameters. This is an improvement to model validation and uncertainty quantification stemming beyond the originally proposed scope of the project. We have introduced a new metric for incorporating the concept of robustness into experiment-based validation of numerical models. This project has accounted for the graduation of two Ph.D. students (Kendra Van Buren and Josh Hegenderfer) and two M.S. students (Matthew Egeberg and Parker Shields). One of the doctoral students is now working in the nuclear engineering field and the other one is a post-doctoral fellow at the Los Alamos National Laboratory. Additionally, two more Ph.D. students (Garrison Stevens and Tunc Kulaksiz) who are working towards graduation have been supported by this project. OBJECTIVES AND ACCOMPLISHMENTS ACHIEVED The project team has successfully formulated a systematic framework for improving the predictive maturity of multi-scale models. Figure 1 illustrates the contribution of each specific objective to the development of this framework. In the following sections, details of the team’s efforts and accomplishments within each objective will be presented. Each of these objectives has been demonstrated on a DOE-relevant problem. Figure 1. Framework for Improving Predictive Maturity of Multi-scale Models Objective 1: To develop an entropy based approach to determine the relative value of each experiment and/or addition of new physics. The project team has investigated the information gain by the addition of each new experiment used for calibration and bias correction of a simulation model. The information gain is measured in terms of the reduction in the uncertainties of the model predictions. We have evaluated the use of various information gain measures based on distance or divergence and entropy concept. Using the meso-scale Visco-Plastic Self Consistent (VPSC) plasticity model for metals, we have investigated the relationship between the information gain and the Predictive Maturity Index (PMI), a quantitative index developed to measure the predictive capabilities of a numerical model. The methods developed for this objective are presented in Report #1. [Hegenderfer J. (2012), “Resource Allocation Framework: Validation of Numerical Models of Complex Engineering Systems Against Physical Experiments,” PhD Dissertation, Clemson University.] Presentation #1 includes a discussion on this objective. We have also investigated the variability of ‘information gain’ through the design domain (i.e. the domain within which the model is intended to be used) in order to identify the experiment settings where maximum information gain occurs and thus guide the experimenters in the selection of the experiment settings. We also investigated if the information gain from each experiment can be improved by intelligently selecting the experiments through Batch Sequential Design (BSD). In BSD, for a given set of initial experiments and a numerical model, a batch of new ‘optimal’ experiments are selected, where the size of the batch as well as the number of batches are defined by the available resources. This work is presented in Report #2 [Atamturktur, S., Williams, B., Edgeberg, M., Unal, C. (2013) “Batch Sequential Design of Optimal Experiments for Improved Predictive Maturity in Physics-based Modeling,” Structural and Multidisciplinary Optimization, Vol. 48, No. 3, pp. 549-569.] We have incorporated the PMI as a BSD selection criteria to further optimize the improvement in predictive maturity during experiment selection. Furthermore, we have modified the approach in calculation of ‘coverage’ of the experiments within the domain of applicability of the model in the PMI to mitigate the overestimation of the coverage using “convex hull” when the experiments are scattered in the four corners of the design space (2D). The incorporation of uncertainty-based information gain measure within the PMI used as an optimality criterion in BSD is currently under way. The application of the PMI-based optimality criterion for BSD procedure on the improved Climb-and-Glide version of the VPSC model is currently being carried out. This work is detailed in Report #1 (Chapters 3 and 4) and is beyond what is originally proposed in the project proposal. Finally, the project team developed a conceptual framework for experiment-based validation and uncertainty quantification of simulation models. This framework extends the framework originally proposed by Unal et al. 2011 from Los Alamos National Laboratory for certification of nuclear fuels by incorporating the decision node for resource allocation. This framework aids in the selection between experimental campaign and code development strictly from the perspective of predictive capability so that the limited resources can be allocated in an efficient manner. This framework builds on and formalizes the concept of trade-off between experimental campaign and code development previously proposed by the project team and that has been applied to PTW model for metals in Report #3 [Atamturktur, S., Hemez, F., Williams, B., Tome, C., Unal, C. (2011) “A Forecasting Metric for Predictive Modeling,” Computers and Structures, Vol. 89, pp. 2377-2387.]. Information gain metric is incorporated within the framework to account for the reduction of uncertainties in the model predictions. The framework is demonstrated on a DOErelevant, fuel-demo problem using the viscoplastic self-consistent (VPSC) code for modeling behavior of metals in Report #4 [Atamturktur S., Hegenderfer J., Williams B., Egeberg M., Lebensohn R. and Unal C., (accepted, in print), “A Resource Allocation Framework for Experiment-Based Validation of Numerical Models,” Journal of Mechanics of Advanced Materials and Structures]. [1] Unal C, Williams B, Hemez F, Atamturktur SH, McClure P. Improved best estimate plus uncertainty methodology, including advanced validation concepts, to license evolving nuclear reactors. Nuclear Engineering and Design 2011; 241: 1813–1833. Objective 2: To investigate the feasibility of multi-scale validation and uncertainty quantification. The second objective of the project involves the development of a Model Validation and Uncertainty Quantification (MVUQ) framework considering limited resources specifically tailored for strongly coupled (multi-scale or multi-physics) models. We have developed a multiscale approach for experiment-based validation and uncertainty quantification taking advantage of both separate-effect experiments conducted within each constituent’s domain to test the validity of the independent constituents in their respective scales and integral-effect experiments executed within the coupled domain to test the validity of the entire coupled system. Our approach takes advantage of partitioned analysis by calibrating and bias-correcting each constituent prior to coupling to reduce the accumulation of errors and uncertainties. Report #5 applies this bias-corrected partitioned analysis method to a DOE-relevant, fuel-demo problem using the meso-scale viscoplastic self-consistent (VPSC) code coupled with a macro-scale finite element model for simulating creep of HCP zirconium. [Stevens, G., Atamturktur, S., Lebensohn, R., Kaschner, G. (to be submitted), “Experiment-based Validation and Uncertainty Quantification of Multi-scale Plasticity Models,” Journal of Mechanics and Physics of Solids.]. A summary of this work is shown in Presentation #2. We have expanded the framework to also incorporate verification aspects and new methodologies for further code development and design of experiments developed by the project team. The framework is discussed in detail in Report #6. [Atamturktur, S. and Stevens, G. (2014), “Uncertainty Inference for Inexact Coupled Numerical Models in Partitioned Analysis,” Proceedings of IX International Conference on Structural Dynamics, Porto, Portugal] and presented in Presentation #3. Specific components of the framework are discussed in Report #7 and Report #2 as follows: Report #7 presents a detailed discussion on one of the components of the framework that is the prioritization of code development efforts. The project team has developed a code prioritization index (CPI) to assess the need and relative improvement in coupled model predictions that can be gained from code development of a constituent model. CPI considers sensitivity analysis, error analysis, and uncertainty analysis of each constituent to ultimately rank the need for code development of each constituent. In this report, the framework is applied to select constituents for further code development in a steel-frame model, specifically focusing on improvement in the modeling of the bolted connections. [Hegenderfer J. and Atamturktur S., (2013), “Prioritization of Code Development Efforts in Partitioned Analysis,” Computer Aided Civil and Infrastructure Engineering (Wiley), Vol. 28, No. 4, pp.289-306]. Report #2 presents a detailed discussion on one of the components of the framework that is the efficient design of calibration experiments, which was developed as part of Objective 1 [Atamturktur, S., Williams, B., Egeberg, M., and Unal, C. (2013), “Batch Sequential Design of Optimal Experiments for Improved Predictive Maturity in Physics-Based Modeling,” Structural and Multidisciplinary Optimization, Vol. 48, No. 3, pp 549-569]. The project team is planning to expand our work on validation of coupled numerical models beyond the original project proposal through the development of a state-aware calibration method. State-aware calibration is an extension of the current state of the art model calibration methods, which implement a black-box approach to infer model bias. These black-box approaches bear no consideration for the sources of bias or its physical meaning. In many numerical models systematic bias may be the result of neglected relationships, be they due to weakly coupled or strongly coupled physical relationships, existing between parameters of the model. The next logical step towards advancing model calibration and validation techniques is decomposing the inference of model bias into a mathematical representation of the nature of missing engineering principles. State-aware calibration is a novel statistical method that considers dependent relationships between parameters, shifting model calibration towards a white-box method. These dependent relationships between parameters may be due to weak coupling or strong coupling, which has not been explicitly modeled. The research plan for implementing state-aware calibration for inferring both weakly and strongly coupled relationships in described in Report #8. Objective 3: To develop a predictive maturity framework to monitor the required amount of experimental information as a function of increased sophistication in the physics of a simulation model. The project team has evaluated the importance of sufficiently exploring the domain of applicability in experiment-based validation of high-consequence modeling and simulation. We have identified characteristics necessary for sufficient coverage. Report #9 presents a discussion on methods for defining coverage and introduces an improved coverage metric [Atamturktur S., Egeberg M., Hemez F., Stevens, G. (2015), "Defining Coverage of an Operational Domain Using a Modified Nearest-Neighbor Metric," Mechanical Systems and Signal Processing, Vol. 50, pp. 349-361]. These methods are also discussed in Presentation #4. We have also incorporated the coverage metric into the design of experiments. The new approach for design of calibration experiments combines the exploration and exploitation based techniques. This method is presented in Report #10 [Atamturktur S., Hegenderfer J., Williams B., Unal, C. (2014), “A Selection Criterion Based on Exploration-Exploitation Approach for Optimal Design of Experiments,” Journal of Engineering Mechanics (ASCE), Vol. 141, No. 1]. The project team has been developing a complexity metric to be implemented for data-aware calibration of numerical models. Data-aware calibration is a concept that suggests the use of an optimally complex model calibration campaign for a given amount of available experiments to achieve the maximum generalizability in calibrated model predictions. In model calibration, formulating a more complex problem is not only vain but it also leads models with reduced predictive capability. In determining the optimum complexity of a calibration campaign, one observes a trade-off between reducing the fitting error and reducing the prediction error. In this work, the team has identified five criteria that should be considered when quantifying complexity. The first two criteria involve the number of calibration parameters identified in the model along with the range within which these parameters are allowed to vary. The model form is also significant for the definition of complexity. The final two criteria are experimental coverage and uncertainties of these experiments. The goal of data-aware calibration is to estimate the optimum calibration complexity based on a given set of experiments. The work completed on this topic is shown in Report #11 [Kulaksiz, T. and Atamturktur, S. (to be submitted) “DataAware Calibration of Numerical Models,” Computers and Structures (Elsevier).] The final component in the project team’s work to improve model calibration and validation methods is the incorporation of robustness to non-probabilistic uncertainty in the input parameters. We have introduced a new metric for incorporating the concept of robustness into experiment-based validation of numerical models. In the context of scientific computing, validation aims to determine the worthiness of a model in regard to its support of critical decision making. This determination of worthiness must occur in the face of unavoidable idealizations in the mathematical representation of the phenomena the model is intended to represent. These models are often parameterized further complicating the validation problem due to the need to determine appropriate parameter values for the imperfect mathematical representations. There exists a need for a method to evaluate the usefulness of models for their intended use given the availability of information. In Report #12 [Shields P. (2013), “Evaluating the Predictive Capability of Numerical Models Considering Robustness to Non-probabilistic Uncertainty in Input Parameters,” MS Thesis, Clemson University.], team has evaluated the agreement between a model’s predictions and associated experiments as well as the robustness of this agreement given imperfections in both the model’s mathematical representation of reality as well as its input parameter values. We complete this evaluation by introducing the concept of a satisfying boundary, which maintains a compact form for proper continuous functions. Deriving the satisfying boundary of a 2dimensional model allows us to visually observe the trade-off between the allowable error in the model predictions and the probability of achieving predictions that indeed satisfy this tolerance for a given uncertainty in model parameters. This evaluation can be repeated for various levels of parameter uncertainty to investigate the aforementioned qualities. This process allows us to observe how the model reacts to uncertainty in its input parameters and thus, make betterinformed decisions using model predictions. Presentation #5 illustrates this method for model validation considering robustness to non-probabilistic uncertainties. PUBLICATIONS RESULTING FROM PROJECT Journal Publications Atamturktur S., Stevens G., Van Buren K. and Wheeler E. (accepted, in print), “Assessing the Trade-offs of Fidelity, Robustness, and Self-Consistency for Model Parameter Identification,”Engineering Computations: International Journal for Computer-Aided Engineering and Software (Emerald). Atamturktur S., Egeberg M., Hemez F., Stevens, G. (2015), "Defining Coverage of an Operational Domain Using a Modified Nearest-Neighbor Metric," Mechanical Systems and Signal Processing, Vol. 50, pp. 349-361 Atamturktur S., Hegenderfer J., Williams B., Unal, C. (2014), “A Selection Criterion Based on Exploration-Exploitation Approach for Optimal Design of Experiments,” Journal of Engineering Mechanics (ASCE), Vol. 141, No. 1 Atamturktur, S., Hegenderfer, J., Williams, B., Egeberg, M., Ricardo, L., and Unal, C., (2014), “A Resource Allocation Framework for Experiment-Based Validation of Numerical Models,” Journal of Mechanics of Advanced Materials and Structures (Taylor & Francis), DOI 10.1080/15376494.2013.828819. Atamturktur, S., Williams, B., Edgeberg, M., Unal, C. (2013) “Batch Sequential Design of Optimal Experiments for Improved Predictive Maturity in Physics-based Modeling,” Structural and Multidisciplinary Optimization, Vol. 48, No. 3, pp. 549-569. Hegenderfer J. and Atamturktur S., (2013), “Prioritization of Code Development Efforts in Partitioned Analysis,” Computer Aided Civil and Infrastructure Engineering (Wiley), Vol. 28, No. 4, pp.289-306. Atamturktur, S., Hemez, F., Williams, B., Tome, C., Unal, C. (2011) “A Forecasting Metric for Predictive Modeling,” Computers and Structures, Vol. 89, pp. 2377-2387. Kulaksiz, T. and Atamturktur, S. (to be submitted) “Data-Aware Calibration of Numerical Models,” Computers and Structures (Elsevier). Stevens, G., Atamturktur, S., Lebensohn, R., Kaschner, G. (to be submitted), “Experiment-based Validation and Uncertainty Quantification of Multi-scale Plasticity Models,” Journal of Mechanics and Physics of Solids. Conference Proceedings Atamturktur, S. and Stevens, G. (2014), “Uncertainty Inference for Inexact Coupled Numerical Models in Partitioned Analysis,” EURODYN IX International Conference on Structural Dynamics, June 30-July 2, Porto, Portugal. Atamturktur, S. and Stevens, G., (2014) “Validation of Strongly Coupled Models: A Framework for Resource Allocation” IMAC XXXII A Conference and Exposition on Structural Dynamics, February 3-6, 2014, Rosen Plaza Hotel, Orlando, FL, USA Egeberg, M., Atamturktur S., and Hemez, F. (2013), “Defining Coverage of a Domain Using a Modified Nearest-Neighbor Metric,” Proceedings of 31st Society of Experimental Mechanics (SEM) International Modal Analysis Conference (IMAC-XXVIII), Orange County, California, USA. Hegenderfer, J., Gillen, A. and Atamturktur S., (2012), “Damage Detection in Steel Structures Using Bayesian Calibration Techniques,” Proceedings of 30th Society of Experimental Mechanics (SEM) International Modal Analysis Conference (IMAC-XXVIII), Jacksonville, Florida, USA. Stevens, G. and Atamturktur, S. (2015), “Experimental Validation and Uncertainty Quantification of Partitioned Models,” NAFEMS World Congress, June 21-24, San Diego, CA. Stevens, G., Atamturktur, S., Lebensohn, R., Kaschner, G., (2014), “Experiment-based Validation and Uncertainty Quantification of Multi-scale Plasticity Models” IMAC XXXII A Conference and Exposition on Structural Dynamics, February 3-6, 2014, Rosen Plaza Hotel, Orlando, FL, USA. Stevens, G., Atamturktur, S., Hegenderfer, J., (2014) “Improving Model Predictions Through Partitioned Analysis” IMAC XXXII A Conference and Exposition on Structural Dynamics, February 3-6, 2014, Rosen Plaza Hotel, Orlando, FL, USA. Theses and Dissertations Egeberg M. (2014), “Optimal Design of Validation Experiments for Calibration and Validation of Complex Numerical Models,” MS Thesis, Clemson University. Shields P. (2013), “Evaluating the Predictive Capability of Numerical Models Considering Robustness to Non-probabilistic Uncertainty in Input Parameters,” MS Thesis, Clemson University. Van Buren K. (2012), “Assuring Robustness against Uncertainty in Predictive Modeling,” PhD Dissertation, Clemson University. Hegenderfer J. (2012), “Resource Allocation Framework: Validation of Numerical Models of Complex Engineering Systems against Physical Experiments,” PhD Dissertation, Clemson University.